PROMPT TUNING

Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards (2022.findings-emnlp)

Challenge:	Existing works on implicit discourse relation recognition focus on syntax features and lack of connectives.
Approach:	They propose a prompt-based path prediction method that integrates the interactive information and intrinsic senses among the hierarchy in IDRR.
Outcome:	The proposed method shows significant improvement against competitive baselines.

Copied to clipboard

Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022.acl-short)

Challenge:	Existing work does not take full advantage of over-parameterized characteristics of large pre-trained language models.
Approach:	They propose a method that uses frozen "thinned" networks to obtain a mixture of rewards and advance the derivative-free prompt learning.
Outcome:	The proposed method outperforms previous gradient-free prompt learning methods and achieves parity with gradient-based counterparts on seven language understanding benchmarks under few-shot settings.

Copied to clipboard

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, Jie Tang

How Does In-Context Learning Help Prompt Tuning? (2024.findings-eacl)

Challenge:	Existing methods of prompt tuning cannot handle hard sequence labeling tasks.
Approach:	They propose to optimize prompt tuning to tune continuous prompts with a frozen language model.
Outcome:	The proposed method matches finetuning with prompt tuning while having only 0.1%-3% tuned parameters.

Copied to clipboard

Simeng Sun, Yang Liu, Dan Iter, Chenguang Zhu, Mohit Iyyer

To Clarify or not to Clarify: A Comparative Analysis of Clarification Classification with Fine-Tuning, Prompt Tuning, and Prompt Engineering (2024.naacl-srw)

Challenge:	a growing number of parameter-efficient adaptation methods are needed to fine-tune large language models.
Approach:	They propose a method that combines prompt tuning and in-context learning to improve prompt tuning by concatenating a natural language demonstration with learned prompt embeddings.
Outcome:	The proposed method outperforms prompt tuning and prompt tuning on five language generation tasks.

Copied to clipboard

Alina Leippert, Tatiana Anikina, Bernd Kiefer, Josef Genabith

Socratic Question Generation: A Novel Dataset, Models, and Evaluation (2023.eacl-main)

Challenge:	Xu et al., 2019) show that pre-trained language model fine-tuning and prompt tuning are better than manual prompt engineering for clarification identification.
Approach:	They propose to use pre-trained language model fine-tuning, prompt tuning and manual prompt engineering to model clarification identification.
Outcome:	The proposed model outperforms pre-trained language model fine-tuning, prompt tuning and manual prompt engineering on the task of clarification identification.

Copied to clipboard

Beng Heng Ang, Sujatha Das Gollapalli, See-Kiong Ng

Text Generation

Challenge:	Socratic questioning is a form of reflective inquiry often employed in education to encourage critical thinking in students.
Approach:	They present a first large dataset of 110K questions, context pairs for Socratic Question Generation.
Outcome:	The proposed model produces realistic, type-sensitive, human-like Socratic questions . authors show that the model can be used in counseling and coaching .

FPT: Feature Prompt Tuning for Few-shot Readability Assessment (2024.naacl-long)

Copied to clipboard

Ziyang Wang, Sanwoo Lee, Hsiu-Yuan Huang, Yunfang Wu

The Power of Prompt Tuning for Low-Resource Semantic Parsing (2022.acl-short)

Challenge:	Prompt-based methods lack crucial linguistic knowledge for readability assessment tasks such as word length, sentence length, and usage of different difficulty-level words.
Approach:	They propose a new prompt-based tuning framework that incorporates linguistic knowledge and a loss function to calibrate the similarity ranking order between categories.
Outcome:	The proposed framework outperforms the large language model gpt-3.5-turbo-16k in most cases.

Copied to clipboard

Nathan Schucher, Siva Reddy, Harm de Vries

Prompt Tuning with Contradictory Intentions for Sarcasm Recognition (2023.eacl-main)

Challenge:	Prompt tuning is an effective method for adapting pre-trained language models to downstream tasks.
Approach:	They propose to use prompt tuning for semantic parsing to map natural language utterances onto formal meaning representations.
Outcome:	The proposed method outperforms the fine-tuned model on low-resource splits of Overnight and TOPv2 on language representations with increasing model scale and target representations.

Copied to clipboard

Yiyi Liu, Ruqing Zhang, Yixing Fan, Jiafeng Guo, Xueqi Cheng

Sentiment Analysis

Challenge:	Recent advances have shown that Pre-trained Language Models (PLMs) can achieve promising performance in many downstream Natural Language Processing (NLP) tasks.
Approach:	They propose to incorporate prior knowledge about contradictory intentions into prompt tuning for sarcasm recognition by mimicking the actual intention by verbalizer engineering.
Outcome:	The proposed model mimics the actual intention by prompt construction and indicates whether the actual intent contradicts the literal content by verbalizer engineering.

Prompt Tuning for Unified Multimodal Pretrained Models (2023.findings-acl)

Copied to clipboard

Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang Zhou

Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering (2024.naacl-short)

Challenge:	Prompt tuning has demonstrated success in natural language pretraining and even vision pretraining.
Approach:	They propose to apply prompt tuning to a unified sequence-to-sequence pretrained model by adding a sequence of learnable embeddings to each layer and finetuning the pretrained models on downstream tasks.
Outcome:	The proposed method outperforms other parameter-efficient tuning methods on multimodal models and is robust against adversarial attacks.

Copied to clipboard

Minji Jung, Soyeon Park, Jeewoo Sul, Yong Suk Choi

TrojFSP: Trojan Insertion in Few-shot Prompt Tuning (2024.naacl-long)

Challenge:	Prompt tuning is an efficient method for initializing pre-trained models . but initialization of prompts is sensitive when the model size is small .
Approach:	They propose a method to measure catastrophic forgetting by analyzing prompts for the first time . they characterize a question answering task based on answer format and prompt initialization .
Outcome:	The proposed approach can help deepen understanding of prompt tuning.

Copied to clipboard

Mengxin Zheng, Jiaqi Xue, Xun Chen, Yanshan Wang, Qian Lou, Lei Jiang

A Dynamic Self-Evolving Extraction System (2026.acl-demo)

Challenge:	Prompt tuning on a few data samples presents security issues, e.g., Trojan attacks.
Approach:	They propose a method to transfer established data poisoning attacks directly to few-shot prompt tuning, a technique to address the poisoned imbalance issue.
Outcome:	The proposed method achieves an ASR of over 99% while maintaining negligible decreases in CDA.

Copied to clipboard

Moin Aminnaseri, Hannah Kim, Estevam Hruschka

Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization (2023.findings-emnlp)

Challenge:	High-quality information extractions often require domain-specific accuracy, up-to-date understanding of specialized taxonomies, and the ability to incorporate emerging jargon and rare outliers.
Approach:	They propose a Dynamic Self-Evolving Extraction and Curation Toolkit which continuously improves as it is used to extract structured information from raw text.
Outcome:	The proposed toolkit continuously improves as it is used in medical, legal, and HR domains.

Copied to clipboard

Kaihang Pan, Juncheng Li, Hongye Song, Jun Lin, Xiaozhong Liu, Siliang Tang

SharPT: Shared Latent Space Prompt Tuning (2023.findings-eacl)

Challenge:	Existing methods for prompt tuning can overfit to few-shot training samples, causing overfitting . authors propose a new framework for prompt learning with supervised meta-learning .
Approach:	They propose a self-supervised meta-prompt learning framework with MEta-gradient Regularization for few-shot generalization that leverages self-recognized meta-learning with a diverse set of meta-tasks to learn a universal prompt initialization using only unlabeled data.
Outcome:	The proposed framework learns a universal prompt initialization for efficient adaptation using only unlabeled data.

Copied to clipboard

Bo Pang, Semih Yavuz, Caiming Xiong, Yingbo Zhou

How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts (2023.findings-emnlp)

Challenge:	Prompt tuning is an efficient method for adapting large language models, but it is difficult and expensive to identify the source task that provides optimal prompts.
Approach:	They propose to learn a shared latent space which captures a set of basis skills from a mixture of source tasks and then transfer them to target tasks.
Outcome:	The proposed method outperforms previous methods on NLI, sentence completion, QA, conference resolution, word sense disambiguation and on various model scales.

Copied to clipboard

Tharindu Kumarage, Paras Sheth, Raha Moraffah, Joshua Garland, Huan Liu

Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts (2022.findings-emnlp)

Challenge:	Existing methods to detect AI-generated text are inadequate, causing misuse of the text.
Approach:	They propose a universal evasive prompt framework that can prompt any PLM to generate “human-like” text that can mislead detectors.
Outcome:	The proposed approach can prompt any PLM to generate “human-like” text that can mislead detectors.

Copied to clipboard

Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, Xipeng Qiu

Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning (2024.eacl-long)

Challenge:	Prompt tuning is parameter-efficient but lags behind other state-of-the-art methods.
Approach:	They propose a parameter-efficient tuning method that only optimizes a soft prompt to adapt PTMs to downstream tasks.
Outcome:	The proposed method is parameter-efficient but lags behind other state-of-the-art methods.

Copied to clipboard

Kaige Xie, Tong Yu, Haoliang Wang, Junda Wu, Handong Zhao, Ruiyi Zhang, Kanak Mahadik, Ani Nenkova, Mark Riedl

Low-Resource NLP

Challenge:	Existing prompt transfer techniques lack consideration for dialogue-specific information.
Approach:	They propose a method which leverages skeleton generation as extra supervision that functions as a medium connecting the distinct source and target task.
Outcome:	The proposed method significantly outperforms baselines on two dialogue summarization benchmarks.

Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis (2022.acl-long)

Copied to clipboard

Hui Wu, Xiaodong Shi

Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples (2023.findings-emnlp)

Challenge:	Existing prompt tuning methods for cross-domain sentiment analysis have been underutilized due to domain discrepancy in the token distributions.
Approach:	They propose a new method to model cross-domain sentiment analysis using pre-trained language models by using soft prompts instead of hard templates.
Outcome:	The proposed method achieves state-of-the-art results on a publicly available sentiment analysis dataset.

Copied to clipboard

Zixuan Ren, Yang Zhao, Chengqing Zong

The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)

Challenge:	Pretrained language models (PLMs) have impressive capabilities in open-ended text generation.
Approach:	They propose a dynamic knowledge-guided informative open-ended text generation approach that utilizes a knowledge graph to help the model generate more contextually related entities and detailed facts.
Outcome:	The proposed approach generates more informative texts than baselines.

Copied to clipboard

Brian Lester, Rami Al-Rfou, Noah Constant

Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation (2022.findings-emnlp)

Challenge:	Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples.
Approach:	They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.
Outcome:	The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters.

Copied to clipboard

Xu Guo, Boyang Li, Han Yu

Prompt Tuning for Discriminative Pre-trained Language Models (2022.findings-acl)

Challenge:	Prompt tuning is a technique for adapting large-scale pretrained language models for downstream tasks.
Approach:	They propose to condition a frozen pretrained language model with soft prompts from data . they propose to use a domain adaptation technique to regularize the decision boundary .
Outcome:	The proposed method outperforms full-model tuning in data-scarce settings by a large margin.

Copied to clipboard

Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang

Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach (2022.findings-acl)

Challenge:	Recent studies have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing tasks.
Approach:	They propose a prompt tuning framework that reformulates NLP tasks into a discriminative language modeling problem.
Outcome:	The proposed framework improves on text classification and question answering tasks and prevents unstable tuning problems in low-resource settings.

Copied to clipboard

Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, Jie Zhou

Frugal Prompting for Dialog Models (2023.findings-emnlp)

Challenge:	Pre-trained language models capture factual knowledge from massive texts . but they are still quite behind the SOTA KGC models in terms of performance .
Approach:	They propose to use open-world assumption to evaluate PLM-based knowledge graph completion models . they propose to convert each triple and its support information into natural prompt sentences .
Outcome:	The proposed model is more accurate under the open-world assumption (OWA) this setting manual checks the correctness of knowledge that is not in KGs.

Copied to clipboard

Bishal Santra, Sakya Basak, Abhinandan De, Manish Gupta, Pawan Goyal

Continued Pretraining for Better Zero- and Few-Shot Promptability (2022.emnlp-main)

Challenge:	Large language models (LLMs) are used in natural language processing tasks with an unrealistic speed and effectiveness.
Approach:	They propose more compact ways of providing dialog history information while ensuring good performance and reducing model’s inference-API costs.
Outcome:	The proposed models have the optimal usable-information density while maintaining good performance and reducing model’s inference-API costs.

Copied to clipboard

Zhaofeng Wu, Robert L Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, Iz Beltagy

Streamlining the Collaborative Chain of Models into A Single Forward Pass in Generation-Based Tasks (2025.findings-acl)

Challenge:	Recent language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters.
Approach:	They propose to use a dedicated pretraining stage to improve promptability in zero-shot settings and few-shot tuning.
Outcome:	The proposed method improves promptability in zero- and few-shot settings, while the existing method yields subpar performance.

Copied to clipboard

Yuanjie Lyu, Chao Zhang, Yuhao Chen, Yong Chen, Tong Xu

Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration (2022.acl-long)

Challenge:	Recent advances in "Chain of Models" approach increase resource demands as each model must be deployed separately.
Approach:	They propose a prompt-tuning method that enables models to share hidden states . they modify input and attention masks during training to eliminate redundant forward passes .
Outcome:	Empirical results show that FTHSS matches the performance of traditional model chains while improving inference efficiency.

Copied to clipboard

Xiwen Liang, Fengda Zhu, Li Lingling, Hang Xu, Xiaodan Liang

Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition (2023.acl-long)

Challenge:	Existing methods of fine-tuning vision-language navigation models require extra human-labeled data and lack self-exploration capabilities in environments.
Approach:	They propose a method that can self-explore environments without human labeling . they use a large-scale cross-modal pretrained model to build an in-domain dataset .
Outcome:	The proposed model can self-explore environments without human labeling without human supervision and generates structured instructions without human intervention.

Copied to clipboard

Haodong Zhao, Ruifang He, Mengnan Xiao, Jing Xu

Noisy Channel Language Model Prompting for Few-Shot Text Classification (2022.acl-long)

Challenge:	Multi-level implicit discourse relation recognition (MIDRR) aims at identifying hierarchical discourse relations among arguments.
Approach:	They propose a prompt-based multi-level implicit discourse relation recognition framework that leverages parameter-efficient prompt tuning to drive inputted arguments to match the pre-trained space.
Outcome:	The proposed framework achieves comparable results on PDTB 2.0 and 3.0 using about 0.1% trainable parameters compared with baselines.

Copied to clipboard

Sewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

Spotlighter: Revisiting Prompt Tuning from a Representative Mining View (2025.findings-emnlp)

Challenge:	Prior work has suggested methods for finding better prompt or scoring of the output from the model.
Approach:	They propose a noisy channel approach for language model prompting in few-shot text classification by in-context demonstration or prompt tuning.
Outcome:	The proposed model outperforms direct models in both demonstration and prompt tuning.

Copied to clipboard

Yutong Gao, Maoyuan Shao, Xinyang Huang, Chuang Zhu, Yu Weng, Xuan Liu, Lijuan Sun, Guoshun Nan

Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models (2022.findings-emnlp)

Challenge:	Spotlighter is a lightweight token-selection framework that enhances accuracy and efficiency in prompt tuning.
Approach:	They propose a token-selection framework that enhances accuracy and efficiency in prompt tuning by preserving only the top-scoring tokens for downstream prediction.
Outcome:	The proposed framework outperforms CLIP by up to 11.19% in harmonic mean accuracy and achieves 0.8K additional FPS, with only 21 extra parameters.

Copied to clipboard

Lifu Tu, Caiming Xiong, Yingbo Zhou

Residual Prompt Tuning: improving prompt tuning with residual reparameterization (2023.findings-acl)

Challenge:	Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks.
Approach:	They do cross-lingual evaluation using prompt tuning and compare it with fine-tuning . prompt tuning achieves much better cross-linguistic transfer than fine- tuning .
Outcome:	The results show that prompt tuning achieves better cross-lingual transfer than fine-tuning across datasets, with only 0.1% to 0.3% tuned parameters.

Copied to clipboard

Anastasiia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, Amjad Almahairi

ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts (2022.emnlp-main)

Challenge:	Prompt tuning is one of the most parameter-efficient approaches for parameter-effective tuning of pre-trained language models.
Approach:	They propose to reparameterize soft prompt embeddings using a shallow network with a residual connection and use it to tune prompt embeds P.
Outcome:	The proposed method outperforms prompt tuning on SuperGLUE, T5-Base and BERT-Bass models and can reduce the prompt length by 10 times without hurting performance.

Copied to clipboard

Akari Asai, Mohammadreza Salehi, Matthew Peters, Hannaneh Hajishirzi

Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL (2024.acl-long)

Challenge:	a new multi-task, parameter-efficient language model tuning method learns to transfer knowledge across different tasks via a mixture of soft prompts.
Approach:	They propose a multi-task, parameter-efficient language model tuning method that uses soft prompts to learn to transfer knowledge across different tasks.
Outcome:	The proposed method outperforms prompt tuning and outperfies or matches fully fine-tuned tuning approaches that use 10 times more parameters.

Copied to clipboard

Yunseon Choi, Sangmin Bae, Seonghyun Ban, Minchan Jeong, Chuheng Zhang, Lei Song, Li Zhao, Jiang Bian, Kee-Eung Kim

Boosting Natural Language Generation from Instructions with Meta-Learning (2022.emnlp-main)

Challenge:	Prompt tuning is an important technique for directing model behaviors and eliciting desired responses.
Approach:	They propose to find optimal prompt tokens using soft Q-learning to optimize models for prompt tuning.
Outcome:	The proposed method improves on baseline prompt tuning, and the results are more natural and interpretable.

Copied to clipboard

Budhaditya Deb, Ahmed Hassan Awadallah, Guoqing Zheng

Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction (2022.acl-long)

Challenge:	Recent work shows that language models trained with multi-task instructional learning (MTIL) can solve diverse NLP tasks in zero-shot settings with improved performance compared to prompt tuning.
Approach:	They propose to adapt meta-learning to MTIL in three directions: 1) Model Agnostic Meta Learning (MAML), 2) Hyper-Network adaptation to generate task specific parameters conditioned on instructions.
Outcome:	The proposed approaches improve over strong baselines in zero-shot settings and are most impactful when the test tasks are strictly zero- shot and are "hard"

Copied to clipboard

Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, Jing Shao

Position Really Matters: Towards a Holistic Approach for Prompt Tuning (2025.findings-naacl)

Challenge:	Using a prompt-based model, we find that event argument extraction is efficient and generalized well to few-shot settings.
Approach:	They propose a model PAIE for event argument extraction using prompt tuning for extractive objectives.
Outcome:	The proposed model can extract arguments with the same role instead of heuristic threshold tuning.

Copied to clipboard

Xianjun Yang, Wei Cheng, Xujiang Zhao, Wenchao Yu, Linda Ruth Petzold, Haifeng Chen

Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization (2023.emnlp-main)

Challenge:	Prompt tuning is effective in extracting knowledge from foundation models, but its effectiveness is uncertain.
Approach:	They propose a parametric prompt tuning strategy that dynamically determines different factors of prompts based on specific tasks or instances.
Outcome:	The proposed approach improves performance across a wide range of tasks including NLP, vision recognition, and vision-language tasks.

Copied to clipboard

Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, Dejing Dou

Uni-Retrieval: A Multi-Style Retrieval Framework for STEM’s Education (2025.acl-long)

Challenge:	Prompt tuning of Large Language Models (LLMs) can incur performance degradation or low training efficiency.
Approach:	They propose a prompt tuning approach with Adaptive Optimization to enable efficient FL of LLMs.
Outcome:	The proposed approach improves performance and efficiency simultaneously and addresses client drift problems on both the device and server sides.

Copied to clipboard

Yanhao Jia, Xinyi Wu, Li Hao, QinglinZhang QinglinZhang, Yuxiao Hu, Shuai Zhao, Wenqi Fan

Low-Resource NLP

Challenge:	Current retrieval models focus on natural text-image retrieval, which is insufficient for STEM education contexts due to ambiguities in the retrieval process.
Approach:	They propose a diverse expression retrieval task tailored to educational scenarios . they extract query style features as prototypes and build a continuously updated Prompt Bank .
Outcome:	The proposed model outperforms existing retrieval models in most retrieval tasks.

FPT: Improving Prompt Tuning Efficiency via Progressive Training (2022.findings-emnlp)

Copied to clipboard

Yufei Huang, Yujia Qin, Huadong Wang, Yichun Yin, Maosong Sun, Zhiyuan Liu, Qun Liu

Enhancing Court View Generation with Knowledge Injection and Guidance (2024.lrec-main)

Challenge:	Recent prompt tuning (PT) has gained increasing attention as a parameter-efficient way of tuning pre-trained language models (PLMs).
Approach:	They propose a prompt tuning algorithm that uses a small-scale partial PLM and progressively expands its depth and width until the full-model size.
Outcome:	The proposed method could save over 30% of training computations while achieving comparable performance.

Copied to clipboard

Ang Li, Yiquan Wu, Yifei Liu, Kun Kuang, Fei Wu, Ming Cai

StablePT : Towards Stable Prompting for Few-shot Learning via Input Separation (2024.findings-emnlp)

Challenge:	Existing methods of natural language generation (NLG) rely on the extensive parameters of pretrained language models (PLMs) but their effectiveness may be compromised by insufficient domain-specific knowledge.
Approach:	They propose a knowledge-injected prompt encoder to incorporate domain knowledge during the training stage, thereby reducing computational overhead.
Outcome:	The proposed approach outperforms established baselines on real-world data in responsivity of claims and in the ability to transfer domain knowledge.

Copied to clipboard

Xiaoming Liu, Chen Liu, Zhaohan Zhang, Chengzhengxu Li, Longtian Wang, Yu Lan, Chao Shen

Learning to Predict Task Transferability via Soft Prompt (2023.emnlp-main)

Challenge:	Existing studies on prompt tuning have shown that language models can be effective few-shot learners with prompting.
Approach:	They propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by prompt initialization.
Outcome:	Experimental results show that the proposed method outperforms state-of-the-art methods by 6.97% in accuracy and reduces the standard deviation by 1.92 on average.

Copied to clipboard

Lingyun Feng

Few-shot Unified Question Answering: Tuning Models or Prompts? (2023.findings-emnlp)

Challenge:	Experimental results show that fine-tuning pretrained language models on helpful intermediate tasks yields further gains.
Approach:	They propose to train an affinity scoring function to predict transferability between tasks by conditioning on task embeddings.
Outcome:	The proposed method efficiently identifies beneficial tasks for transfer learning.

Copied to clipboard

Srijan Bansal, Semih Yavuz, Bo Pang, Meghana Bhat, Yingbo Zhou

Low-Resource NLP

Challenge:	Question-answering (QA) tasks investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks.
Approach:	They propose to use model and prompt tuning for unified QA in a low-resource setting to overcome drawbacks of unified models.
Outcome:	The proposed model and prompt tuning paradigms outperform model tuning in a few-shot setting with a good initialization and achieve a significant performance boost from pre-training in 'low-resource' setting.

StablePrompt : Automatic Prompt Tuning using Reinforcement Learning for Large Language Model (2024.emnlp-main)

Copied to clipboard

Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, Junmo Kim

GPS: Genetic Prompt Search for Efficient Few-Shot Learning (2022.emnlp-main)

Challenge:	Recent advances in large language models have made it difficult to find appropriate prompts for tasks with multiple input-output formats.
Approach:	They propose a prompt tuning method based on reinforcement learning (RL) they propose an anchor model and an extension for generating input-dependent prompts.
Outcome:	The proposed method outperforms existing methods on a variety of tasks and achieves State-of-the-art performance across diverse types and sizes of LLMs.

Copied to clipboard

Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Wang Yanggang, Haiyu Li, Zhilin Yang

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)

Challenge:	Pretrained language models are often finetuned for downstream tasks, which has been shown to improve performance over non-pretrained models.
Approach:	They propose a genetic algorithm to automatically search for the best prompt for few-shot learning with pretrained language models by gradient-free algorithm.
Outcome:	Experiments on diverse datasets show that the proposed method outperforms manual prompts by 2.6 points.

Copied to clipboard

Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu

PPT: Pre-trained Prompt Tuning for Few-shot Learning (2022.acl-long)

Challenge:	Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement.
Approach:	They propose a method that involves tuning a small set of soft prompts for pre-trained language models.
Outcome:	The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark.

Copied to clipboard

Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang

Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts (2023.findings-emnlp)

Challenge:	Prompt tuning for pre-trained language models has shown remarkable performance . however, prompt tuning is still not fully explored .
Approach:	They propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization.
Outcome:	The proposed framework outperforms full-model tuning under full-data and few-shot learning settings.

Copied to clipboard

Muling Wu, Wenhao Liu, Jianhan Xu, Changze Lv, Zixuan Ling, Tianlong Li, Longtao Huang, Xiaoqing Zheng, Xuanjing Huang

Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models (2025.coling-main)

Challenge:	Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models.
Approach:	They propose a token-wise prompt tuning method that uses a bank of finer-grained soft prompt tokens to generate an instance-dependent prompt.
Outcome:	The proposed method performs far better than full parameter fine-tuned models and achieves state-of-the-art by tuning only 0.035% parameters on 14 datasets.

Copied to clipboard

Sofia Jamil, Bollampalli Areen Reddy, Raghvendra Kumar, Sriparna Saha, Joseph K. J, Koustava Goswami

Multimodality

Challenge:	Poems are a distinct form of literature, with meanings that transcend beyond the literal words.
Approach:	They propose a framework to generate images that visually represent the meanings of poems using prompt tuning and a PoeKey algorithm to extract emotions, visual elements, and themes from poems.
Outcome:	The proposed framework generates images that visually represent the meanings of poems and their images.

Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)

Copied to clipboard

Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova

Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning (2023.acl-long)

Challenge:	Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling .
Approach:	They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods .
Outcome:	The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds .

Copied to clipboard

Tianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu, Xuanjing Huang

Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022.emnlp-main)

Challenge:	Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks.
Approach:	They propose to combine pre-trained modules with pre-trains to boost prompt tuning for few-shot learning.
Outcome:	The proposed model outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot learning settings.

Copied to clipboard

Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, Noah Constant

Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models (2024.lrec-main)

Challenge:	generative multilingual models fine-tuned on English forget to generate non-English data when labeled data is only available in English . generative models fine tuned on English fail to generate multilingual summarization tasks when labeling data is available in other languages .
Approach:	They propose to use prompt tuning to overcome catastrophic forgetting in a generative task in . they assume a strict setting with no parallel data or machine translation .
Outcome:	The proposed method can overcome catastrophic forgetting to enable zero-shot cross-lingual generation.

Copied to clipboard

Sixing Yu, Juan Pablo Munoz, Ali Jannesari

Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting (2023.emnlp-main)

Challenge:	Foundation Models (FMs) have demonstrated success in a wide range of applications, but their optimization often requires access to sensitive data.
Approach:	They propose a framework that combines FMs and Federated Learning to enable privacy-preserving and collaborative learning across multiple end-users.
Outcome:	The proposed framework combines benefits of FMs and Federated Learning (FL) it enables privacy-preserving and collaborative learning across multiple end-users.

Copied to clipboard

Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, Jilin Chen

Global and Local Hierarchical Prompt Tuning Framework for Multi-level Implicit Discourse Relation Recognition (2024.lrec-main)

Challenge:	Existing studies on diversity in large language models focus on the understudied class of fairness and inclusion concern in LLMs.
Approach:	They propose a technique to measure diversity in generated responses along people and culture axes by collective-critique and self-voting.
Outcome:	The proposed approach outperforms baseline methods and human evaluations with human and automated evaluations.

Copied to clipboard

Lei Zeng, Ruifang He, Haowen Sun, Jing Xu, Chang Liu, Bo Wang

SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning (2023.emnlp-main)

Challenge:	Recent methods to recognize hierarchical discourse relations without explicit connectives are inefficient and ignore the utilization of the output probability distribution information of the verbalizer.
Approach:	They propose a global and local hierarchical prompt tuning framework which leverages top-up propagated probability as the global hierarchy to inject it into multi-level verbalizer.
Outcome:	The proposed framework achieves competitive results on two benchmacks.

Copied to clipboard

Wei Zhu, Ming Tan

XPrompt: Exploring the Extreme of Prompt Tuning (2022.emnlp-main)

Challenge:	Prompt tuning is a method of pre-trained models that optimizes the prompt to adapt to downstream tasks.
Approach:	They propose a framework that learns to select the proper prompt layers by inserting a probabilistic gate at each intermediate layer.
Outcome:	The proposed framework can perform better than the state-of-the-art prompt tuning frameworks on ten benchmark datasets.

Copied to clipboard

Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, Dawei Song

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models (2022.emnlp-main)

Challenge:	Prompt tuning learns soft prompts to condition pre-trained Language Models for performing downstream tasks in a parameter-efficient manner.
Approach:	They propose a Prompt tuning model with an eXtremely small scale that learns soft prompts to condition the frozen Pre-trained Language Models for performing downstream tasks in a parameter-efficient manner.
Outcome:	The proposed model outperforms the vanilla Prompt-Tuning and can significantly improve across tasks and model scales.

Copied to clipboard

Yuan Yao, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

Finding Skill Neurons in Pre-trained Transformer-based Language Models (2022.emnlp-main)

Challenge:	Recent advances on self-supervised learning have led to powerful vision-language pre-training models that achieve state-of-the-art performance on a wide range of cross-modal tasks.
Approach:	They propose a vision-language pre-training framework that reformulates discretized object positions and language in a unified language modeling framework.
Outcome:	The proposed model improves performance on position-sensitive vision-language (VL) tasks and also improves on position insensitive tasks.

Copied to clipboard

Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, Juanzi Li

Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024.emnlp-main)

Challenge:	Pre-trained language models have demonstrated superior performance on various natural language processing tasks.
Approach:	They find that after prompt tuning, some neurons encode task-specific skills . they also show that skill neurons are most likely generated in pre-training .
Outcome:	The neurons are highly predictive of task labels after prompt tuning for specific tasks.

Copied to clipboard

Ming Li, Jike Zhong, Chenxin Li, Liuzhuozheng Li, Nie Lin, Masashi Sugiyama

Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt (2023.findings-emnlp)

Challenge:	Recent advances in fine-tuning Vision-Language Models have seen the success of prompt tuning and adapter tuning.
Approach:	They propose a method to fine-tune CLIP without introducing any overhead of extra parameters.
Outcome:	The proposed method improves CLIP by 7.27% average harmonic mean accuracy.

Copied to clipboard

Seonghyeon Ye, Joel Jang, Doyoung Kim, Yongrae Jo, Minjoon Seo

Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation (2025.emnlp-main)

Challenge:	Recent studies show that adding a instruction tuning stage to training large language models can improve zero-shot task generalization.
Approach:	They propose a method that retrieves promptspecific source prompt embeddings from training instances . they train soft prompt embeds for each prompt through prompt tuning and store the samples .
Outcome:	The proposed method outperforms hard prompts on unseen tasks by 2.39% points and outperformed 10 out of 11 datasets.

Copied to clipboard

Ziniu Zhang, Zhenshuo Zhang, Dongyue Li, Lu Wang, Jennifer Dy, Hongyang R. Zhang

PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer (2023.emnlp-main)

Challenge:	Existing methods to select demonstration examples for in-context learning are based on token embeddings.
Approach:	They propose an algorithm to select demonstration examples for in-context learning of a query set . they use gradients of the output taken in the input embedding space to estimate model outputs .
Outcome:	The proposed algorithm outperforms existing methods based on token embeddings by 11% . it scales up subset selection that would otherwise run full inference by 37.7 on models with 34 billion parameters .

Copied to clipboard

Lichang Chen, Jiuhai Chen, Heng Huang, Minhao Cheng

Unlocking Black-Box Prompt Tuning Efficiency via Zeroth-Order Optimization (2024.findings-emnlp)

Challenge:	Existing prompt tuning methods have training instability issues due to large variance of scores . existing prompt tuning algorithms have training stability issues due a slight change of input data .
Approach:	They propose an algorithm that smooths the loss landscape of vanilla prompt tuning by perturbation-based regularizers.
Outcome:	The proposed method improves the state-of-the-art prompt tuning methods by 1.94% and 2.34% on SuperGLUE and FewGLUE benchmarks.

Copied to clipboard

Heshen Zhan, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun

MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering (2025.findings-acl)

Challenge:	Prompt optimization is an important technique for adapting Large Language Models (LLMs) to specific tasks.
Approach:	They propose a zeroth-order approach which enables efficient prompt tuning solely via inference APIs.
Outcome:	The proposed approach outperforms existing black-box prompt tuning methods in terms of performance and convergence speed.

Copied to clipboard

Shuo Yang, Caren Han, Siwen Luo, Eduard Hovy

Multimodality

Challenge:	Existing Large Vision-Language Models (LVLMs) lack integrated commonsense knowledge . lack of integrated common knowledge limits their robustness and accuracy in VQA .
Approach:	They propose a framework to enhance multimodal inference by integrating commonsense reasoning.
Outcome:	MAGIC-VQA improves comprehensive benchmark datasets, surpassing existing models in tasks requiring advanced commonsense reasoning.

Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers (2023.findings-emnlp)

Copied to clipboard

Weng Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Jiahua Liu, Tao Li, Yuxiao Dong, Jie Tang

An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)

Challenge:	Prompt tuning is a technique that updates few parameters in pre-trained models for language understanding and generation tasks.
Approach:	They propose to leverage prompt tuning for neural text retrieval to improve generalization and cross-domain generalization.
Outcome:	The proposed approach can mitigate the two issues faced by fine-tuning retrieval methods and improve the out-of-domain zero-shot generalization of the retrieval models.

Copied to clipboard

Zhongbin Xie, Thomas Lukasiewicz

SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts (2023.emnlp-main)

Challenge:	Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems.
Approach:	They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation.
Outcome:	The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias.

Copied to clipboard

Joon-Young Choi, Junho Kim, Jun-Hyung Park, Wing-Lam Mok, SangKeun Lee

Decomposed Prompt Tuning via Low-Rank Reparameterization (2023.findings-emnlp)

Challenge:	Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models.
Approach:	They propose a prompt tuning method that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced by longer soft prompt.
Outcome:	The proposed method outperforms baseline methods while preserving memory usage.

Copied to clipboard

Yao Xiao, Lu Xu, Jiaxi Li, Wei Lu, Xiaoli Li

Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding (2023.emnlp-main)

Challenge:	Pre-trained language models have achieved remarkable performance on various tasks.
Approach:	They propose a decomposed prompt tuning approach that utilizes low-rank matrices to initialize the soft prompt.
Outcome:	The proposed method significantly reduces the number of trainable parameters while maintaining effectiveness.

Copied to clipboard

Caoyun Fan, Jidong Tian, Yitian Li, Wenqing Chen, Hao He, Yaohui Jin

Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)

Challenge:	Chain-of-Thought (CoT) is a technique that guides large language models to decompose complex tasks into multi-step reasoning processes.
Approach:	They propose a two-step reasoning framework based on prompt tuning to implement step-by-step thinking for MLMs on NLU tasks.
Outcome:	The proposed framework outperforms baselines and achieves state-of-the-art performance on two NLU tasks.

Copied to clipboard

Po-Heng Chen, Yun-Nung Chen

LGSA: Label Geometry Structuring and Aligning for Hierarchical Text Classification (2026.acl-long)

Challenge:	Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks.
Approach:	They propose a method that uses soft-prompt tuning to tune for language adaptation . prompt tuning outperforms continuously trained baselines on two benchmarks .
Outcome:	The proposed approach outperforms baselines on two text classification benchmarks while utilizing 0.28% of tuned parameters.

Copied to clipboard

Shuai Zhang, Weibo Xu, Jiahao Nie, Kecheng Huang

Information Retrieval

Challenge:	Existing hierarchical text classification methods use prompt tuning or contrastive learning to implicitly learn label embeddings for classification, but this method fails to model hierarchy-aware geometric relations among labels.
Approach:	They propose a two-stage framework that transforms the label hierarchy from an implicit prior into an explicit embedding by using a general orthogonal frame.
Outcome:	The proposed framework outperforms existing state-of-the-art methods on three real-world HTC datasets.

RePrompT: Recurrent Prompt Tuning for Integrating Structured EHR Encoders with Large Language Models (2026.findings-acl)

Copied to clipboard

Arya Hadizadeh Moghaddam, Drew Ross, Mohsen Nayebi Kerdabadi, Dongjie Wang, Zijun Yao

Generalizable Prompt Tuning for Audio-Language Models via Semantic Expansion (2026.findings-acl)

Challenge:	Large Language Models (LLMs) have shown promising results for mining EHRs . translating time-stamped sequences into plain text can obscure both temporal structure and code identities, weakening the ability to capture code co-occurrence and longitudinal regularities.
Approach:	They propose a time-aware LLM framework that integrates structured EHR encoders through prompt tuning without modifying underlying architectures.
Outcome:	Experiments on MIMIC-III and MIMIC IV show that RePrompT outperforms both EHR-based and LLM-based baselines across multiple clinical prediction tasks.

Copied to clipboard

Jaehyuk Jang, Wonjun Lee, Kangwook Ko, Changick Kim